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g! Abstract 

Traditional Byzantine resilient algorithms use 2f + 1 vertex disjoint paths to ensure message delivery 
in the presence of up to / Byzantine nodes. The question of how these paths are identified is related to 
. the fundamental problem of topology discovery. 

Distributed algorithms for topology discovery cope with a never ending task, dealing with frequent 
changes in the network topology and unpredictable transient faults. Therefore, algorithms for topology 
discovery should be self-stabilizing to ensure convergence of the topology information following any 
such unpredictable sequence of events. We present the first such algorithm that can cope with Byzantine 
nodes. Starting in an arbitrary global state, and in the presence of / Byzantine nodes, each node is 
eventually aware of all the other non-Byzantine nodes and their connecting communication links. 

Using the topology information, nodes can, for example, route messages across the network and 
deliver messages from one end user to another. We present the first deterministic, cryptographic - 
assumptions-free, self-stabilizing, Byzantine-resilient algorithms for network topology discovery and 
\ end-to-end message delivery. We also consider the task of r-neighborhood discovery for the case in 

\Q ■ which r and the degree of nodes are bounded by constants. The use of r-neighborhood discovery facili- 

, tates polynomial time, communication and space solutions for the above tasks. 

00 1 The obtained algorithms can be used to authenticate parties, in particular during the establishment 

of private secrets, thus forming public key schemes that are resistant to man-in-the-middle attacks of the 
compromised Byzantine nodes. A polynomial and efficient end-to-end algorithm that is based on the 
established private secrets can be employed in between periodical re-establishments of the secrets. 
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1 Introduction 

Self-stabilizing Byzantine resilient topology discovery is a fundamental distributed task that enables com- 
munication among parties in the network even if some of the components are compromised by an adversary. 
Such topology discovery is becoming extremely important nowadays where countries main infrastructures, 
such as the electrical smart-grid, water supply networks and intelligent transportation systems are subject 
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to cyber-attacks. Self-stabilizing Byzantine resilient algorithms naturally cope with mobile attacks [e.g., 
flfitl . Whenever the set of compromised components is fixed (or dynamic, but small) during a period that 
suffice for convergence of the algorithm the system starts demonstrating useful behavior following the con- 
vergence. For example, consider the case in which nodes of the smart-grid are constantly compromised by 
an adversary while local recovery techniques, such as local node reset and/or refresh, ensure the recovery of 
a compromised node after a bounded time. Once the current compromised set does not imply a partition of 
the communication graph, the distributed control of the smart grid automatically recovers. Self-stabilizing 
Byzantine resilient algorithms for topology discovery and message delivery are important for systems that 
have to cope with unanticipated transient violations of the assumptions that the algorithms are based upon, 
such as unanticipated violation of the upper number of compromised nodes and unanticipated transmission 
interferences that is beyond the error correction code capabilities. 

The dynamic and difficult-to-predict nature of electrical smart-grid and intelligent transportation systems 
give rise to many fault-tolerance issues and require efficient solutions. Such networks are subject to transient 
faults due to hardware/software temporal malfunctions or short-lived violations of the assumed settings for 
the location and state of their nodes. Fault-tolerant systems that are self-stabilizing [5] can recover after 
the occurrence of transient faults, which can drive the system to an arbitrary system state. The system 
designers consider all configurations as possible configurations from which the system is started. The self- 
stabilization design criteria liberate the system designer from dealing with specific fault scenarios, risking 
neglecting some scenarios, and having to address each fault scenario separately. 

We also consider Byzantine faults that address the possibility of a node to be compromised by an adver- 
sary and/or to run a corrupted program, rather than merely assuming that they start in an arbitrary local state. 
Byzantine components may behave arbitrarily (selfishly, or even maliciously) as message senders and/or as 
relaying nodes. For example, Byzantine nodes may block messages, selective omit messages, redirect the 
route of messages, playback messages, or modify messages. Any system behavior is possible, when all (or 
one third or more of) the nodes are Byzantine nodes. Thus, the number of Byzantine nodes, /, is usually 
restricted to be less than one third of the nodes 

The task of r -neighborhood network discovery allows each node to know the set of nodes that are at 
most r hops away from it in the communication network. Moreover, the task provides information about the 
communication links attached to these nodes. The task topology discovery considers knowledge regarding 
the node's entire connected component. The r-neighborhood network discovery and network topology 
discovery tasks are identical when r is the diameter of the communication graph. 

This work presents the first deterministic self-stabilizing algorithms for r-neighborhood discovery in the 
presence of Byzantine nodes. We assume that every r-neighborhood cannot be partitioned by the Byzantine 
nodes. In particular, we assume the existence of at least 2/ + 1 vertex disjoint paths in the r-neighborhood, 
between any two non-Byzantine nodes, where at most / Byzantine nodes are present in the r-neighborhood, 
rather than in the entire network. Q Note that by the self-stabilizing nature of our algorithms, recovery is 
guaranteed after a temporal violation of the above assumption. When r is defined to be the diameter of the 
communication graph, our assumptions are equivalent to the standard assumption for Byzantine agreement 
in general (rather than only complete) communication graphs. In particular the standard assumption is that 
2/ + 1 vertex disjoint paths exist and are known (see e.g., jl3ll ) while we present distributed algorithms to 
find these paths starting in an arbitrary state. 

Related work. Self-stabilizing algorithms for finding vertex disjoint paths for at most two paths between 

1 Section [4] considers cases in which r and the node degree, A, are constants. For these case, we have 0(n) disjoint r- 
neighborhoods. Each of these (disjoint) r-neighborhoods may have up to / Byzantine nodes, and yet the above assumptions, 
about at least 2/ + 1 vertex disjoint paths in the r-neighborhood, hold. 
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any pair of nodes, and for all vertex disjoint paths in anonymous mesh networks appear in [Q]] and in [11], 
respectively. We propose self-stabilizing Byzantine resilient procedures for finding / + 1 vertex disjoint 
paths in 2/ + 1-connected graphs. In |9|], the authors study the problem of spanning tree construction in 
the presence of Byzantine nodes. Nesterenko and Tixeuil II 1 5f1 presented a deterministic non-stabilizing 
algorithm for topology discovery in the presence of Byzantine nodes. The authors do not consider the 
automatic recovery implied by the self-stabilization property. [[Awerbuch and Sipser [3] consider algorithms 
that were designed for synchronous static network and give topology update as an example. They show 
how to use such algorithms in asynchronous dynamic networks. Unfortunately, their scheme starts from a 
consistent state and cannot cope with transient faults or Byzantine.]] 

Byzantine gossip 10,0, Jll 12, 13] and Byzantine Broadcast [JsL 03] consider the dissemination of 
information in the presence of Byzantine nodes rather than self-stabilizing topology discovery. Non-self- 
stabilizing Byzantine resilient gossip in the presence of one selfish node is considered in J3,|l3]- In El, the 
authors study oblivious deterministic gossip algorithms for multi-channel radio networks with a malicious 
adversary. They assume that the adversary can disrupt one channel per round, preventing communication on 
that channel. In |]4], the authors consider probabilistic gossip mechanisms for reducing the redundant trans- 
missions of flooding algorithms. They present several protocols that exploit local connectivity to adaptively 
correct propagation failures and protect against Byzantine attacks. Probabilistic gossip mechanisms in the 
context of recommendations and social networks are considered in 111 Oil . In jl4l the authors consider rules 
for avoiding a combinatorial explosion in (non-self-stabilizing) gossip protocol. Note that deterministic and 
self-stabilizing solutions are not presented in l2Lfllil [loT,[l2lll4ll . 

Drabkin et al. 180 consider non-self-stabilizing broadcast protocols that overcome Byzantine failures 
by using digital signatures, message signature gossiping, and failure detectors. Our deterministic self- 
stabilizing algorithm merely use the topological properties of the communication graph to ensure that mes- 
sages dropped or modified by Byzantine nodes will be detected, and retransmitted in a way that guarantees 
correct delivery to the application layer. A non-self-stabilizing broadcasting algorithm is considered in II 1711 . 
The authors assume the restricted case in which links and nodes of a communication network are subject to 
Byzantine failures, and that faults are distributed randomly and independently. 

Our contribution. We present two cryptographic-assumptions-free yet secure algorithms that are deter- 
ministic, self-stabilizing and Byzantine resilient. 

We start by showing the existence of deterministic, self-stabilizing, Byzantine resilient algorithms for 
network topology discovery and end-to-end message delivery. [[The algorithms convergence time is in 
0[n). They take in to account every possible path and requiring bounded (yet exponential) memory and 
bounded (yet exponential) communication costs.]] Therefore, we also consider the task of r-neighborhood 
discovery, where r is a constant. We assume that if the r-neighborhood of a node has / Byzantine 
nodes, there are 2/ + 1 vertex independent paths between the node and any non-Byzantine node in its 
r-neighborhood. The obtained r-neighborhood discovery requires polynomial memory and communica- 
tion costs and supports deterministic, self-stabilizing, Byzantine resilient algorithm for end-to-end message 
delivery across the network. [[Unlike topology update, the proposed end-to-end message delivery algo- 
rithm establishes message exchange synchronization between end-users that is based on message reception 
acknowledgments. ]] 

Document structure. Settings and requirements appear in Section |2] The self-stabilizing Byzantine 
resilient distributed algorithm for topology discovery is presented in Section [3] The end-to-end communi- 
cation algorithm appears in Section 0] Extensions and concluding remarks appear in Section [5] Detailed 
proofs appear in the Appendix and in |7J]. 
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2 Preliminaries 



We consider settings of a standard asynchronous system [cf. |5fl. The system consists of a set, N = {pi} 
of communicating entities, chosen from a set P, which we call nodes. The upper bound on the number of 
nodes in the system is n = \P\. Each node has a unique identifier. Sometime we refer to a set, P \ N, of 
nonexisting nodes that a false indication on their existence can be recorded in the system. A node pi can 
directly communicate with its neighbors, Ni C N. The system can be represented by a network of directly 
communicating nodes, G = (N,E), named the communication graph, where E = {(pi,Pj) G N x N : 
Pj G Ni}. We denote AVs set of indices by indices (N^) = {m : p m G Nk} and A^'s set of edges by 
edges (Nj) = {pj} x Nj. 

The r-neighborhood of a node pi G N is the connected component that includes pi and all nodes that 
can be reached from pi by a path of length r or less. The r-neighborhood version of the algorithm for 
network topology discovery considers communication graphs in which the number of neighbors of a node 
Pi is bounded by a constant A. Hence, when both the neighborhood radius, r, and the node degree A are 
constants the number of nodes in the r-neighborhood is also bounded by a constant, namely by [[0(A r+1 ).]] 

We model the communication channel, queueij, from node pi to node pj G Ni as a FIFO queuing list 
of the messages that pi has sent to pj and pj is about to receive. When pi sends message m, the operation 
send inserts a copy of m to every queueij, such that pj G A^. We assume that the number of messages in 
transit, i.e., stored in queueij, is at most capacity. Once m arrives, pj executes receive and m is dequeued. 

We assume that pi is completely aware of Ni, as in |15l| . In particular, we assume that the identity of 
the sending node is known to the receiving one. In the context of the studied problem, we say that node 
Pi G N is correct if it reports on its genuine neighborhood, Ni. A Byzantine node, pb G N, is a node 
that can send arbitrarily corrupted messages. Byzantine nodes can introduce new messages and modify 
or omit messages that pass through them. This way they can, e.g., disinform correct nodes about their 
neighborhoods, or about the neighborhood of other correct nodes, or the path through which messages 
travel, to name a very few specific misleading actions that Byzantine nodes may exhibit. We denote by 
C and B the set of correct, and respectively, Byzantine nodes. We assume that |B| = /, the identity of 
the nodes in B is unknown to the nodes in C. Nevertheless, B is fixed throughout the considered execution 
segment. These execution segments are long enough for convergence and then for obtaining sufficient useful 
work. We assume that between any pair of correct nodes there are at least 2/ + 1 vertex disjoints paths. We 
denote by G c = (C, E n C x C) the correct graph induced by the set of correct nodes. 

Self-stabilizing algorithms never terminate (see fl). The non-termination property can be easily identi- 
fied in the code of a self-stabilizing algorithm: the code is usually a do forever loop that contains commu- 
nication operations with the neighbors. An iteration is said to be complete if it starts in the loop's first line 
and ends at the last (regardless of whether it enters branches). 

Every node, pi, executes a program that is a sequence of (atomic) steps. For ease of description, we 
assume the interleaving model where steps are executed atomically, a single step at any given time. An 
input event can either be the receipt of a message or a periodic timer going off triggering pi to send. 
Note that the system is totally asynchronous and the (non-fixed) node processing rates are irrelevant to the 
correctness proof. 

The state Sj of a node pi consists of the value of all the variables of the node (including the set of all 
incoming communication channels, {queue j^\pj G Ni}). The execution of a step in the algorithm can 
change the state of a node. The term (system) configuration is used for a tuple of the form (si, S2, • • • , s n ), 
where each Sj is the state of node pi (including messages in transit for pi). We define an execution E = 
c[0], a[0], c[l], a[l], ... as an alternating sequence of system configurations c[x] and steps a[x\, such that each 
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configuration c[x + 1] (except the initial configuration c[0]) is obtained from the preceding configuration c[x] 
by the execution of the step a[x]. We often associate the notation of a step with its executing node pi using 
a subscript, e.g., cij. An execution R (run) is fair if every correct node, pi G C, executes a step infinitely 
often in R. Time (e.g. needed for convergence) is measured by the number of asynchronous rounds, where 
the first asynchronous round is the minimal prefix of the execution in which every node takes at least one 
step. The second asynchronous round is the first asynchronous round in the suffix of the run that follows the 
first asynchronous round, and so on. The message complexity (e.g. needed for convergence) is the number 
of messages measured in the specific case of synchronous execution. 

We define the system's task by a set of executions called legal executions (LE) in which the task's 
requirements hold. A configuration c is a safe configuration for an algorithm and the task of LE provided 
that any execution that starts in c is a legal execution (belongs to LE). An algorithm is self-stabilizing with 
relation to the task LE when every infinite execution of the algorithm reaches a safe configuration with 
relation to the algorithm and the task. 

3 Topology Discovery 

The topology discovery is based on accumulating messages from vertex disjoint paths. Each message con- 
tains an ordered list of nodes it passed so far, starting in a source node, and a neighborhood, which is the set 
of nodes, claimed to be directly connected to the source. 

Each node periodically sends a message to each neighbor. The message sent contains the local 
topology, a source i and an empty path. The arrival of a message m to pi triggers an insert of m to 
informedTopologyi and a consistency test of the content of informedTopologyi. The consistency test 
results in storing local topologies for which there are enough independent evidence in a result array. The 
result array is initialized just prior to the consistency test. The consistency test of pi iterates over each node 
Pk such that, pk appears in at least one of the messages stored in informedTopologyi. For each such node 
Pk, node pi checks whether there are at least / + 1 messages from the same source node that have mutually 
vertex disjoint paths and report on the same neighborhood. The neighborhood of each such p^., that has at 
least / + 1 vertex disjoint paths with identical neighborhood, is accumulated in Result[k]. Moreover, the 
total number of paths [[that]] relayed this neighborhood is kept in Count[k\. 

We note that there may still be nodes pj a ke £ P\(Cl)B), for which there is an entry Result[fake]. For 
example, informedTopology may contain / messages, all originated from different Byzantine nodes, and 
a message m! that appears in the initial configuration and supports the (false) neighborhood the Byzantine 
messages refer to. These f + I messages can contain mutually vertex disjoint paths, and thus during the 
consistency test, a result will be found for Result[fake\. We show that during the next computations, the 
message m! will be identified and ignored. 

The Result set should include two reports for each (undirected) edge; the two nodes that are attached to 
the edge, each send a report. Hence, Result includes a set of directed (report) edges. The term contradicting 
edge is needed when examining the Result set consistency. 

Definition 1 (Contradicting edges) Given two nodes, Pi,pj G P, we say that the edge (pi,Pj) is contra- 
dicting with the set Neighborhoodj C edges(Nj), if (pi,Pj) Neighbor hood j. 

Following the consistency test, pi examines the Result array for contradictions. Node pi checks the 
path of each message m G informedTopologyi with source p r , neighborhood neighbor hood r and Path r . 
If every edge (p s ,Pj) on the path appears in Result[s] and Result[j], then we move to the next message. 
Otherwise, we found a fake supporter, and therefore we reduce Count[r] by one. In case the resulting 
Count[r] is smaller than / + 1, we nullify the r'th entry of the Result array. Once all messages were 



5 



processed, the Result array consisting of the (confirmed) local topologies is the output. At the end 
forwards the arriving message m to each neighbor that does not appear in the path of m. The message sent 
by pi includes the node from which m arrived as part of the path m. 

The pseudocode appears in Algorithm Q] In every iteration of the infinite loop, pi starts to compute 
its preliminary topology view by calling ComputeResults in line [2] Then, every node p^ in the queue 
InformedTopology, node pi goes over the messages in the queue from head to bottom. While iterat- 
ing the queue, for every message m with source pf., neighborhood -ZV& and visited path Path^, Pi inserts 
Pathk to opinion[Nk], see line[l8j After inserting, pi checks if there is a neighborhood Neigu for which 
opinion[N eigk] contains at least [[/ + 1]] vertex disjoint paths, see line [19] When such a neighborhood 
is found, it is stored in the Result array (line [191. In line [20l pi stores the number of vertex disjoint paths 
relayed messages that contained the selected neighborhood for pk- After computing an initial view of the 
topology, in line pi removes non-existing nodes from the computed topology. For every message m in 
InformedTopology, node pi aims at validating its visited path. In line|24j pi checks if there exists a node 
Pk whose neighborhood contradicts the visited path of m. If such a node exists, pi decreases the associated 
entry in the Count array (linel25T>. This decrease may cause Count[r] to be smaller than / + 1, in this case 
Pi considers p^ to be fake and deletes the local topology of p^ from Result[r] (line [26]). 

Upon receiving a message m, node pi inserts the message to the queue, in case it does not already exist, 
and just moves it to the top of the queue in case it does. The node pi now needs to relay the message pi 
got to all neighbors that are not on the message visited path (line [9]>. When sending, pi also attaches the 
identifier of the node, from which the message was received, to the visited path of the message. 
Algorithm's correctness proof. We now prove that within a linear amount of asynchronous rounds, 
the system stabilizes and every output is legal. The proof considers an arbitrary starting configuration with 
arbitrary messages in transit that could be actually in the communication channel or already stored in pj 's 
message queue and will be forwarded in the next steps of pj. Each message in transit that traverse correct 
nodes can be forwarded within less than C(|C|) asynchronous rounds. Note that any message that traverses 
Byzantine nodes and arrives to a correct node that has at least one Byzantine node in its paths. The reason 
is that the correct neighbor to the last Byzantine in the path lists the Byzantine node when forwarding the 
message. Thus, / is at most the number of messages that encode vertex disjoint paths from a certain source 
that are initiated or corrupted by a Byzantine node. Since there are at least / + 1 vertex disjoint paths with 
no Byzantine nodes from any source pk to any node pi and since pk repeatedly sends messages to all nodes 
on all possible paths, pi receives at least / + 1 messages from p^ with vertex disjoint paths. 

The usage of the FIFO queue and the repeated send operations of pk ensure that the most recent / + 1 
messages with vertex disjoint paths in InformedTopology queue are uncorrupted messages. Namely, 
misleading messages that were present in the initial configuration will be pushed to appear below the new 
/ + 1 uncorrupted messages. Thus, each node pi eventually has the local topology of each correct node 
(stored in the Resulti array). The opposite is however not correct as local topologies of non-existing nodes 
may still appear in the result array. For example, InformedTopologyi may include in the first configuration 
/ + 1 messages with vertex disjoint paths for a non-existing node. 

Since after ComputeResults we know the correct neighborhood of each correct node p^, we may try to 
ensure the validity of all messages. For every message that encodes a non-existing source node, there must 
be a node pi on the message path, such that p£ is correct and p£S neighbor is non-existing, this is true since 
Pi itself is correct. Thus, we may identify these messages and ignore them. Furthermore, no valid messages 
are ignored because of this validity check. 

We also note that, since we assume that the nodes of the system are a subset of P. The size of the queue 
InformedTopology is bounded. Next, we bound the amount of memory of a node. The details of the 
correctness and convergence proofs appear in the Appendix and in 0]. 
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Algorithm 1: Topology discovery, code for node p$ 



Input: N eighborhoodi. The ids of the nodes with which node p t can communicate directly; 
Output: ConfirmedTopology C P X P: Discovered topology, which is represent by a directed edge set; 
Variable InformedTopology : Queue, see Figure[T] topological messages, (node, neighborhood, path); 
Function: jV odeDisj ointP 'aths(S): Test S = {(node, neighborhood, path)} to encode at least / + 1 vertex disjoint paths; 
Function: PathContradicts Neighbor hood(k, Neighborhood^, path): Test that there is no node pj £ JV for which there is an edge 

(pi,Pj) in the message's visited path, path C P x JV, such that (pk,Pj) is contradicting with Neighborhoodi; 
while true do 

Result <— ComputeResultsQ 
let Result <— RemoveContradictions(Result) 
RemoveGarbage(Result) 

ConfirmedTopology <— ConfirmedTopology U (Up gp Result[k]) 
foreach pi £ JV; do send(i, Neighborhoodi, 0) to pi 

Upon Receive ((£, Neighborhood^, VisitedPathi)) from p.,; 
begin 

Insert(pi, Neighborhoodi, VisitedPathi U {j}) 

foreach pi £ Ni do if k £ VisitedPathi then send(p£, Neighborhoodi, VisitedPathi U {j}) to pi 

Procedure: I nsert(k, Neighbor hood^, VisitedPathi); 
begin 

if 3m = (£, Neighborhoodi, VisitedPathi) £ InformedTopology : (£, Neighborhoodi, VisitedPathi) 
k Neighborhood^, VisitedPathi) then Inf ormedT opology.MoveT oHeadim) 
else if Pfc £ JV A Neighborhood^ C indices(N) A VisitedPathi C indices(N) then 
InformedTopology. Insert((k, Neighbor hood k , VisitedPathi)) 

13 Function: ComputeResultsQ; 
begin 

foreach p fc £ P : (k, Neighborhood k ,VisitedPath k ) £ InformedTopology do 

let (FirstDisjointPathsFound, Message, opinion[]) •<— (false, InformedTopology. IteratorQ, [0]) 
while Message. hasN ext() do 

(I, Neighborhoodi, VisitedPathi) Message. Next() 

iff = fc then opinion[Neighborhoodi].Insert((£, Neighborhoodi, VisitedPathi)) 
if FirstDisjointPathsFound = false A N odeDisj ointPaths(opinion[N eighborhoodi]) then 
I (Result[k], FirstDisjointPathsFound) <— (Neighborhoodi, true) 

Count[k] <— opinion[Result [k.SizeOfQ]] 

return Result 

22 Function: RemoveC 'ontradictions(Result); 
begin 

foreach (r,Neighborhood r ,VisitedPath r ) £ InformedTopology do 

if 3pfc £ P : PathContradictsN eighborhood(pk, Result[k],VisitedPath r ) = true then 
if JV eighborhoodr = Resu/J[r] then Count[r] <— Count[r] — 1 
if Coimi[r] < / then Pesu2J[r] <- 

return Result 

Procedure: RemoveGarbage(Result); 
begin 

foreach p fc £ JV do 

foreach m = (fc, JV eighborhoodi, VisitedPathi) £ InformedTopology : 

{k} U Neighborhoodi U VisitedPathi 2 P V Jn / ormedT 'apology .1 s After (m, opinion[k][Result[k]]) do 
7n/ormedTopoio(;j/.Pe7?io^e(m) 



Lemma 1 (Bounded memory) Letpi he a correct node. At any time, there are at mostn-2 2n messages 
in In f ormedT opologyanyi, where n = |P| araJ C(|P| log(|P|)) jj message size. 

r-neighborhood discovery. Algorithm Q] demonstrates the existence of a deterministic self-stabilizing 
Byzantine resilient algorithm for topology discovery. Lemma[T]shows that the memory costs are high when 
the entire system topology is to be discovered. We note that one may consider the task of r-neighborhood 
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• Insert(m): Insert item m to the head of the queue. 

• Remove(M essagem): Remove item m from the queue. 

• IteratorQ: Returns an pointer for iterating over the queue's elements by the order in which they reside in the queue. 

• HasNextQ: Tests whether the Iterator is at the end of the queue. 

• NextQ Returns the next element to iterate over. 

• SizeOf() Returns the number of elements in the calling set. 

• MoveToHead(m): Move item m to the head of the queue. 

• IsAfter(m, S): Test that item m is after the items m 1 S S, where S is a set of items in the queue. 

Figure 1: Queue: general purpose data structure for queuing items, and its operation list. 

discovery. Recall that in the r-neighborhood discovery task, it is assumed that every r-neighborhood cannot 
be partitioned by Byzantine nodes. Therefore, it is sufficient to constrain the maximal path length in line [9] 
The correctness proof of the algorithm for the r-neighborhood discovery follows similar arguments to the 
correctness proof of Algorithm Q] 

4 End-to-End Delivery 

We use the discovered network topology to design a self-stabilizing Byzantine resilient algorithm for the 
transport layer protocol. Namely, using the repeatedly collected topology information for implementing end- 
to-end communication between (not necessarily neighboring) nodes. In this context, we face the challenge 
of finding / + 1 correct vertex disjoint paths and the need to propose efficient solutions for different system 
settings. 

The value of ConfirmedTopology is a set of directed edges (pi,Pj). An undirected edge is approved 
if both (pi,Pj) and (j)j,Pi) appear in ConfirmedTopology. An edge is said to be suspected, whenever 
only one edge (in one direction) appears in ConfirmedTopology. The sender has to choose 2/ + 1 vertex 
independent paths to the receiver. If there exists at least one such set of paths then the sender can safely use 
them to communicate with the receiver (similar to Algorithm [T]). However, the collected topology may not 
include even one such set of 2/ + 1 vertex independent paths. The reason is that / of the paths that should 
appear in the collected topology may be controlled by Byzantine nodes. Namely, the information about at 
least one edge in each such path may not arrive to the sender. 

We propose three procedures for overcoming this difficulty in different system setting. The first proce- 
dure assumes / is a constant. Thus, the sender may apply the following procedure for selecting a set of vertex 
disjoint paths Paths, that contains / + 1 correct paths. For each possible choice of / nodes pi,p%, . . - Pf in 
the system, the sender computes a new graph G which is the result of removing px,p2, ■ ■ - Pf, from G ou t, 
the graph defined by the collected topology. The sender now computes a set V of vertex disjoint paths, 
where \V\ = f + 1, if such a set exists. For each such set V, the sender relays the current message on all 
paths in V. First we show that this procedure only sends message through a polynomial number of paths. 
There are 0{n^) possibilities for choosing / nodes from the system. Thus, 0(n*) sets of paths are com- 
puted, and since / is a constant, this number is polynomial. Moreover, each such set contains at most / + 1 
paths, because only computes a set V of size / + 1. Thus, in total, the sender sends the message on at 
most a polynomial number of paths. We now show that this procedure ensures that the message is sent on a 
sufficient amount of correct paths i.e., / + 1. Consider the permutation in which the set of / chosen nodes 
actually contains the set of Byzantine nodes in the system. Thus G contains only correct nodes. Further- 
more, at least / + 1 paths that were present in G ou t are still present in G, since we removed / nodes. Hence, 
in G, there are at least / + 1 correct vertex disjoint paths. As stated, the sender chooses a set of paths of size 
f + I. Each of these paths is correct, and therefore the sender sends the message on at least f + I correct 
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vertex disjoint paths as needed. 

The second procedure assumes that r and A are both constants. The sender sends the message over all 
possible paths to the receiver. This is feasible only when considering r-neighborhoods, rather than the entire 
connected component, where the neighborhood radius, r, and the node degree A are constants. Next, we 
present a polynomial solution for the case in which /, r and A are not constants, assuming that Byzantine 
nodes are not directly connected. 

The third procedure assumes that Byzantine nodes cannot be immediate neighbors and that all neighbors 
of a given Byzantine node refer to the Byzantine with the same identifier. Our polynomial cost solution 
considers the (extended) graph, G ex t, that includes all the edges in confirmedTopology and suspicious 
edges, see Definition |2l 

Definition 2 (Suspicious edges) Given three nodes, Pi,Pj,Pk 6 P> we say that node pi considers the undi- 
rected edge (pk,Pj) suspicious, if the edge appears as a directed edge in ConfirmedTopologyi for only 
one direction, e.g., (pj,Pk)- 

The extended graph, G ex t, may contain fake edges that Byzantine nodes reports on their existence. Never- 
theless, G ex t includes all the correct paths of the communication graph, G. Therefore, the 2/ + 1 vertex 
disjoint paths that exists in G also exists in G ex t- These 2/ + 1 paths facilitate our polynomial cost solution. 

The sender uses the chosen paths to repeatedly forward the message m that should arrive to the receiver. 
The sender uses a label to identify the messages. Roughly speaking, the receiver deliver a message received 
at least c • n + 1 consecutive times from / + 1 vertex independent paths (according to the path carried in the 
message). Once the receiver delivers a message to the network layer, the receiver starts to repeatedly send 
acknowledgments with the label of the delivered message over 2/ + 1 vertex disjoint paths. In addition, the 
receiver also restarts its counters and the log of received messages upon a message delivery to the network 
layer. Similarly the sender count acknowledgments to the current label used, when the sender receives at 
least c • n + 1 acknowledgments on a set of / + 1 vertex disjoint paths, the sender fetches the next message 
from the network layer, changes the label and starts to send the new message. We note that stalling from 
an arbitrary configuration, the sender eventually fetches a message from the network layer. This is obvious 
since if the sender is sending the same message forever, then the receiver counters on / + 1 paths must 
exceed c-n + 1. From this point the receiver sends acknowledgments with the correct label forever ensuring 
that the sender fetches the next message. 

The pseudocode of the algorithm appears in Algorithm |2] In every iteration of the infinite loop, pi 
fetches a message (line [3]). Following the fetch, pi prepares the label for the next message (line 0]). Once 
the label is ready, pi starts sending the message over 2/ + 1 vertex disjoint messages which pi calculates 
in the procedure ByzantineFaultToleranceSend(Message). When pi gets enough acknowledgments 
regarding the current message (see line|5]), pi stops sending the current message and fetches another message. 

Upon receiving a message m, node pi checks in line |7] whether pi is the destination of the message. If 
not, pi forwards the message to the next node on the intended path of the message, not forgetting to update 
the visited path. If however pi is the destination of the message, pi checks the type of the message in 
line [TUJ If the type of the message is Data then (in line [TTb pi inserts the message payload and label to the 
part of the data structure associated with the message source, i.e., the sender, and the message visited path. 
In line 27, node pi checks whether 2/ + 1 vertex disjoint paths relayed the message at least capacity ■ n + 1 
times, where capacity is an upper bound on the number of messages in transit over a communication link. 
If so, pi delivers the message to the above layer (line l20l ). clears the entire data structure and finally sends 
acknowledgments on 2/ + 1 vertex disjoint paths until a new message is confirmed. Moreover, in line [21] 
we signal that we are ready to receive a new message. If the type of the message is ACK, we act almost as 
when the message is of type Data. When the condition in line [18] holds, we signal that the message was 
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confirmed at the receiver by setting Approved to be true, in line [18] 

Correctness proof. Let us consider three labels, 0, 1, and 2 that are used by the sender in a round robin 
fashion. Whenever at least c • n + 1 identical messages arrive at the receiver with the same label on each 
of / + 1 vertex independent paths, the receiver delivers them, nullify the counters, empty queues and send 
acknowledges with the label of the delivered message over 2f + 1 vertex-disjoint paths (cf. line[T3T>. The 
sender clears counters and queues whenever the sender changes label. 

First we prove that the sender fetches infinitely often, by assuming it is not and proving that eventually 
the receiver sends acknowledgments with the label used by the sender. Hence, the sender must fetch (see 
Lemma [T3T>. Then in between the second and the fourth fetch of any four successive fetches, where without 
the loss of generality, the first fetch is with label 0, the second with 1, the third with label 2 and the fourth 
with the receiver clears its counter and the last fetched message in this sequence that is with label is later 
delivered. 

Following the fetch of each of the first three messages and before the next one, the sender must count 
c • n + 1 acknowledgments with the current label that the sender uses to send, namely with 0, 1 and 2. 
Since the sender reset the counters when changing the sending label to 1, the receiver must send at least 
one acknowledgment with label 1 and then with label 2, following the corresponding fetches. Thus, the 
receiver must clear its counters at least once following the second fetch and before the fourth fetch and then 
start sending acknowledgments with label 2. After clearing the counters by the receiver and starting sending 
acknowledgments with label 2 a message with label that is next to be sent, must be delivered and no other 
message can be counted as arriving at least c • n + 1 times through / + 1 vertex-disjoint paths. Detailed 
proof appears in the Appendix and in |2D. 

Note that the code of Algorithm [2] considers only one possible pair of source and destination. A many- 
source to many-destination version of this algorithm can simply use a separate instantiation of this algorithm 
for each pair of source and destination. 

5 Extensions and Conclusions 

As extension, we suggest to combine the algorithms for r-neighborhood network discovery and the end-to- 
end capabilities in order to allow the use of end-to-end message delivery within the r-neighborhoods. These 
two algorithms can be used by the nodes, under reasonable node density assumptions, for discovering their 
r-neighborhoods and then extending the scope of their end-to-end capabilities beyond their r-neighborhood, 
as we sketch next. We instruct further remote nodes to relay topology information, and in this way collect 
information on remote neighborhoods. One can consider an algorithm for studying specific remote neigh- 
borhood that are defined, for example, by their geographic region, assuming the usage of GPS inputs; a 
specific direction and distance from the topology exploring node defines the exploration goal. The algo- 
rithm nominates 2/ + 1 nodes in the specific direction to return further information towards the desired 
direction. The sender uses end-to-end communication to the current 2/ + 1 nodes in the front of the current 
exploration, asking them for their r-neighborhood, chooses a new set of 2/ + 1 nodes for forming a new 
front. It then instructs each of the current nodes in the current front to communicate with each node in the 
chosen new front, to nominate the new front nodes to form the exploration front. 

To ensure stabilization, this interactive process of remote information collection should never stop. 
Whenever the current collection process investigates beyond the closest r-neighborhood, we concurrently 
start a new collection process in a pipeline fashion. The output is the result of the last finalized collection 
process. Thus, having a correct output after the first time a complete topology investigation is finalized. 
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In this work we presented two deterministic, self-stabilizing Byzantine-resilience algorithms for topol- 
ogy discovery and end-to-end message delivery. We have also considered an algorithm for discovering 
r-neighborhood in polynomial time, communication and space. Lastly, we mentioned a possible extension 
for exploring and communicating with remote r-neighborhoods using polynomial resources as well. 

The obtained end-to-end capabilities can be used for communicating the public keys of parties and 
establish private keys, in spite of / corrupted nodes that may try to conduct man-in-the-middle attacks, an 
attack that the classical Public key infrastructure (PKI) does not cope with. Once private keys are established 
encrypted messages can be forwarded over any specific / + 1 node independent paths, one of which must 
be Byzantine free. The Byzantine free path will forward the encrypted message to the receiver while all 
corrupted messages will be discarded. Since our system should be self-stabilizing, the common private 
secret should be re-established periodically. 
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Algorithm 2: Self-stabilizing Byzantine resilient end-to-end delivery, code for node p^. 

Interface: FetchMessageQ: Get a new message from the upper layer. We denote by InputM essageQueue the unbounded queue of 

all messages that are to be delivered to the destination; 
Interface: Deliver M essage(Source, Message): Deliver an arriving message to the higher layer. We denote by 

OutputM essageQueue the unbounded queue of all messages that are to be delivered to the higher layer. We assume that it 

always contains at least the last message inserted to it; 
Input: ConfirmedTopology: The discovered topology, which is represent by a set of directed edges included in P X P, see 
Algorithm Q] 

Data Structure: Transport layer messages: (Source, Destination, VisitedPath, IntentedPath, ARQLabel, Type, Payload), 
where Source is the sending node, Destination is the target node, VisitedPath is the actual relay path, 
IntentedPath is the planned relay path, ARQ Label is the sequence number of the stop-and-wait ARQ protocol, and 
Type 6 {Data, ACK} message type, where DATA and ACK are constant; 
Field: Payload: the message data; 

Variable Message: the current message being sent; 

Variable ReceivedM essages[j] [Path] : queue of pj 's messages that were relayed over path Path (see Figure[T}; 

Variable Confirmations^] [Path] : queue of Pj's message acknowledgments that were relayed over path Path (see Figure[TJ; 

Variable label: the current sequence number of the stop-and-wait ARQ protocol; 

Variable Approved: A Boolean variable indicating whether Message was accepted at the destination; 

Function: jV odeDisj ointP 'aths(S): Test S, a set of paths, to encode at least / + 1 vertex disjoint paths; 

Function: FloodedP 'ath(M essageQueue, m) : Test whether m is encoded by the first capacity ■ n + 1 messages in 

MessageQueue, where capacity is an upper bound on the number of messages in transit over a communication link.; 
Function: SuspiciousEdgesQ : Get the set of suspicious edges; 

Function: getDisj ointP 'aths(T 'apology , Source, Destination) : Get a set of / + 1 vertex disjoint paths between source and 

destination in the graph induced by Topology.; 
Function: ClearQueue(Source) : Delete all data in ReceivedM essages[Source][*]; 
Function: Clear AckQueue(Destination) : Delete all data in Confirmations[Destination] [*]; 
l while true do 

Clear AckQueue{M essage. Destination) 
Message <— FetchMessageQ 
label <— label + 1 modulo 3 

while Approved = false do ByzantineFaultToleranceSend(Message) 

Upon Receive (msg) From pj ; 
begin 

if msg . Destination ^ i then 

msg. VisitedPath <— msg. VisitedPath U {j} 
send(msg) 

else if msg.Type = Data then 

ReceivedM essages[msg. Source] [msg. VisitedPath].insert((msg. Payload, msg. ARQ Label)) 
if 3m G ReceivedMessages[msg.Source][*] : Paths = {Path : 

FloodedPath(ReceivedMessages[msg.Source][Path],m)} A NodeDisjointPaths(Paths) A 
msg. source = m. source then 

13 Con firm(msg. Source, m. ARQLabel, m. Payload) 

14 NewMesssage = true 



15 

16 



17 
18 



else if msg.Type = ACK then 

it label = msg. ARQLabel then 
\__ Con firmations[msg. Source] [msg. VisitedPath] .insert({msg. Payload, msg . ARQ Label)) 

let Paths <— {Path : FloodedPath(Confirmations[msg.Source][Path], (msg. Payload, msg. ARQLabel))} 
if N odeDisj ointPaths(Paths) then Approved = true 



19 Function: Confirm(Source, ARQLabel, Payload); 
begin 

20 if CurrentLabel ARQLabel then Deliver M essage(Source, Payload) 

21 (CurrLbl, NewMessage) <— (ARQLbl, false) 

22 ClearQueue(Source) 

23 while NewMessage = false do ByzantineFaultToleranceSend((Source, ARQLabel, ACK, Payload)) 

24 Function: ByzantineF aultT oleranceSend(Destination, ARQLabel, Type, Payload); 
begin 

25 let Paths <— getDisjointPaths(ConfirmedTopology U SuspiciousEdges(),i, Destination) 

26 foreach Path £ Patlhs do send((i, Destination, 0, Path, ARQ LabehpType, Payload)) to Rrst(Path) 
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A Correctness of Algorithm CD 



Lemma[T](Bounded memory) Let G C be a correct node. At any time, there are at most n ■ 2 2n messages 
in InformedTopologyanyi, where n = \P\ and 0{\P\ logdi-*!)) is the message size. 
Proof. The queue InformedTopologyanyi, is made up of messages in the form 

(node, neighborhood, visitedpath). All nodes that appear in the message, i.e., in the first, second 
or third entry of the tuple are in N. The first entry, i.e. the node name is one of n possibilities. The second 
and third entries are subsets of N. Thus each of them has 2" possibilities. In total there can be at most 
2 n ■ 2" • n messages in every InformedTopologyanyi. ■ 

Definition [3] specifies the requirements of the network topology discovery task. Definition 0] considers 
correct paths and Definition [5] considers uncorrupted graph topology messages. 

Definition 3 (Legal output) Given correct node pi G C, we say that pi 's output is legal, if it encodes 
graph G output = {V out ,E out ): (1) C C V^ C C U B C N, and (2) (E n (C x C)) C E out C 
(E n (C x C)) U (B x (C U B)) C N x N. 

Definition 4 (Correct path) We say path C N is a correct one if all its nodes are correct, i.e., path C C. 
Definition 5 (Valid message) Algorithm [7] we refer a message m = 

(k, Neighborhoodk, VisitedPathk) as a valid message when: (1) pk € C a«<i VisitedPathk encodes a 
correct path in the communication graph, G, that starts in p^, and (2) Neighbor hood & = indices (N^). 

Lemma|2]shows that eventually correct paths do not relay non valid messages. Namely, invalid messages 
can only exist as the result of: (1) Byzantine interventions that corrupt messages, or (2) transient faults, 
which occur only prior to the arbitrary starting configuration considered. H 

Lemma 2 (Eventually valid messages) Let R be a fair execution of Algorithm\7]that starts in an arbitrary 
configuration. Within 0(\CUB\) asynchronous rounds, the system reaches a configuration after which only 
valid messages are relayed on correct paths. 

Proof. Let c E R be the starting configuration. Suppose that c includes an invalid message, m = 
(£, Neighborhoode,VisitedPathe), in transit between correct nodes. The lemma is obviously correct 
for the case that m is relayed by Byzantine nodes during the first 0(\C U B\) asynchronous rounds of R. 
Therefore, we consider only the correct paths, path, over which m is relayed during the first C(|C U B\) 
asynchronous rounds of R. We show that, within 0(\C U B\) asynchronous rounds, no correct node in path 
relays m. 

Let Pj,Pi € path be correct neighbors on the correct path. Suppose that in c, message m 
is in transit from pj to pi. Upon the arrival of message m to pi (line [7]), pi sends m% = 
(£, Neighborhoode, VisitedPathk U {j}} to any neighbor p^ € path on the path for which p^ G JVj A k 
VisitedPathe, see line [9] 

Node pi adds pj's identifier to m's visited path VisitedPathk, see line [9j The same argument holds 
for any correct neighbors, p'j,p'j G path when pj sends message m'- to the next node in path, node p\. 

2 This is a common way to argue about self-stabilization, we consider executions that start in an arbitrary configuration that 
follows the last transient fault, recalling that if additional transient faults occur a new arbitrary configuration is reached from which 
automatic convergence starts. 
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Therefore, within \path \ VisitedPathi | asynchronous rounds, it holds that fl (path \ VisitedPathi) = 

{p'pp'iY 

Note that p\ makes sure that VisitedPath' e does not encode loops, i.e., p^ g VisitedPath! g , see line[9] 
Therefore, node p\ does not relay message m' to pj.. ■ 

Definition [6] considers queues that their recent valid messages encode at least / + 1 vertex disjoint paths. 
Moreover, the invalid ones encode at most / such paths. 

Definition 6 (Valid queue) Let Pi,pk £ C be two correct nodes. We say that pi 's queue, 
InformedTopologyi, is valid (with respect to p^) whenever there is a prefix, V alidlnf ormationi^, 
of messages in the queue InformedTopologyi, such that: (I) there is a subset, Valid = 
{rri£ = (k, Neighborhood/^, VisitedPathi) : mi is valid} C V alidlnf or motion i^, for which the 
set {VisitedPathi} encodes at least f + 1 vertex disjoint paths, and (2) the set, Invalid = {mi = 
(k, Neighborhoods, VisitedPathi) : mi is invalid} C Validlnformationi ^, for which the set 
{VisitedPathi} encodes at most f vertex disjoint paths. 

Claim [3] shows that, within C(|C|) asynchronous rounds, correct paths propagate valid messages. 

Claim 3 Let path C C be a correct path from pi to p^. Suppose that mi = (i, Ni, 0} is a (valid) message 
that pi sends, see line® Within 0(\path\) asynchronous rounds, message mi is relayed on path, and arrives 
at pk as ?7J ■ = {i, Ni,path). Namely, path is m\'s visited path. 

Proof. Let c G R be the first configuration that follows the start of m^'s propagation in path. I.e., c is 
the configuration that immediately follows the step in which node pi sends m, by executing line [6] Let 
p r ,Pj € path be two correct neighbors on the path. Without the loss of generality, suppose that node pi 
sends message directly to node p r , i.e., in c, node p r is just about to receive mi. The proof arguments 
hold also when assuming that pj sends message mj = (i, Ni, {r}} to the next node in path. Thus, generality 
is not lost. 

We show that, within one asynchronous round, p r sends m r to pj. Upon the arrival of message mi 
to p r (line [7]), node p r sends the message m r to any neighbor, such as pj, for which pj G N r A r $ 
VisitedPathi = 0, see line [9] Since the same argument holds when pj sends mj to the next node in path, 
we have that within \path\ asynchronous rounds, m\ is delivered to node p^. 

□ 

Lemma @] shows that queues get to become valid. 

Lemma 4 (Eventually valid queues) Let R be a fair execution of Algorithm\J\that starts in an arbitrary 
configuration andpi,p^ S C be any pair of correct nodes. The system reaches a configuration in which the 
queue, InformedTopologyi, is valid (with respect to pk), within 0(\C U B\) asynchronous rounds. 
Proof. Let c G R be a configuration achieved in Lemma [2] within 0(\C U B\) asynchronous rounds. We 
show that within 0(\C U B\) asynchronous rounds after c, the system reaches a configuration in which 
InformedTopologyi, is valid (with respect to pk), see Definition^ 

In configuration c, all messages in transit on correct paths are valid, see Lemma [2] Thus, the only 
messages entering InformedTopologyi are either valid or have passed through Byzantine nodes. Denote 
mbarrier to be the top message the queue InformedTopologyi. Moreover, V alidlnf ormationi^ includes 
all the messages in InformedTopologyi, that are between the queue's head and mbarrier- 

We show that condition (1) of Definition [6] holds. There are 2/ + 1 vertex disjoint paths between pi and 
Pk- At most / nodes are Byzantine and thus, there are at least / + 1 vertex disjoint paths between pi and 
Pk that are correct. By Claim[3]within C(|C|) asynchronous rounds, a valid message, m^, is received on all 
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/ + 1 (correct) vertex disjoint paths. Message rrik is inserted to InformedTopologyi after configuration c. 
Therefore, m\~ is in front of nibarrier- Hence, the set Valid = {me = (£, Neighbor hoode, VisitedPathi) : 
me is valid} C Validlnformatiorii^ contains at least /+ 1 valid messages whose respective visited paths, 
VisitedPathe, are vertex disjoint. 

We show that condition (2) of Definition [6] holds. Any invalid messages, m^, that is sent after configu- 
ration c, must go through a Byzantine node, see Lemma|2] 

Claim 5 Suppose that message m is relayed through a Byzantine node after configuration c, then in any 
following configuration, while m is still in transit, there is a Byzantine node in the visited path. 

Proof. Observe the first correct node p^ after the last Byzantine node b on m's path, p^ is correct, thus it 
inserts b to the visited path, b is the last on the path and so the visited path must contain it until end of transit 
or passing through a different Byzantine. □ 

Each such Byzantine node is recorded in the message path, see Claim [5] Since there are at most / 
Byzantine nodes, there could be at most / such messages with vertex disjoint paths. This completes the 
proof condition (2) and the lemma. ■ 

Lemma |7] shows that correct information gets confirmed, and requires Definition [7] 

Definition 7 (Message confirmation) We say that message mi = (k, Neighborhood}-, VisitedPathe.) is 
confirmed (by node pi) when Neighbor hoodk C ConfirmedTopologyi. 

Lemma 6 (Eventually confirmed messages) Let R be a fair execution of Algorithm [7J that starts in 
an arbitrary configuration and pt,Pk £ C be any pair of correct nodes. Within 0(\C U B\) 
asynchronous rounds, the system reaches a configuration after which the fact that message mi = 
(k, Neighborhood):, VisitedPath^) is confirmed, implies that Neighborhood^ = indices(Ne). 
Proof. Let c € R be the first configuration in which InformedTopologyi is a valid queue and node pi 
completes a full iteration of the do forever loop that starts in line Q] By Lemma @J the system reaches c 
within 0(\C U B\) asynchronous rounds. 

We how that in configuration c, the array Resulti satisfies that Resulti[k\ = indices(Ne). We go 
through the computation of Result in lines [2]to |4] 

• C omputeResults(), line [2] Let Resi[k] = indices(N' i ) be ComputeResultsQ's re- 
turn value with respect to node pk- We show that Resi[k] = indices(Ne). Moreover, we show 
that the neighborhood that will be found will be that which is represented in Valid = {me = 
{k, Neighbor hoodk, VisitedPathe) ■ me is valid} C V alidlnf ormatiorii^- 

We recall that the set {VisitedPathe} encodes at least / + 1 disjoint paths. Also in the prefix 
Validlnformatiorii^k one can not find / + 1 invalid messages with vertex disjoint messages; See Def- 
inition [6] 

The function must choose the message containing the neighborhood N eighbor hoodk- Otherwise, we 
have chosen a different neighborhood for k, say Neighborhood^ ^ N eighborhoodk = indices(Nk). That 
is, at the time of checking line [19] with neighborhood N eighbor hoode = Neighborhood^, there were at 
least / + 1 vertex disjoint paths in opinion[N eighbor hoode]. This is in contradiction to condition (2) of 
Definition [6] Moreover in line [20l it holds Count[k] > / + 1, since at least all the correct paths were 
counted. 

• RemoveContradictionsi), line [3] Let Resi = ComputeResultsi) and 
ResRemoveContradictionsi = RemoveContradictions(Resi) (line [3). We show that 
ResRemoveContradictionsi[r] = indices(N r ). The function RemoveContradictionsi) modifies 
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Resi[r] only in line|26]by nullifying it whenever Count[r] < f. We demonstrate that, for any correct path 
VisitedPathk, there exists nope for which PathContradictsNeighborhood(pe, Resi[£], VisitedPathk) 
= true, which is the condition in linel24l 

We explain that there is no node pe and a contradicting edge (pj,pe) with the set Resi[£]. By the 
assumption that VisitedPathk is correct and that node pi G VisitedPathk, we have that pi G C is correct. 
Thus Resi[l] = indices(Ne), see previous item of this claim on ComputeResultsQ. VisitedPathk is 
correct, and therefore (pj,pe) must be in VisitedPathk- 

• RemoveGarbageQ, line 01 This procedure does not modify Resi = 

RemoveContradictions(ComputeResultsQ). We have shown that Resulti [k] = indices(Nk). Thus, 
only the correct neighborhood is confirmed for every correct node pk- ■ 

Lemma|7]shows that eventually there are no fake nodes. 

Lemma 7 (Eventually no fake nodes) Let R he a fair execution of Algorithm\l\that starts in an arbitrary 
configuration, pj G N be any node, andpi G P\{C\JB) be a node that is not included in the communication 
graph, G. Within 0(\CUB\) asynchronous rounds, the system reaches a configuration after which (pj,pi) ^ 
ConfirmedTopologyi 

Proof. Let c G R be the configuration reached within 0(\C U B\) asynchronous rounds according to 
Lemma[6] For any correct node, pi G C, we show that in c, the execution of RemoveContradictionsQ 
results in Count i[£] < f and nullifies Resulti[£]. 

We start by showing that for every path p that relays a message which encodes the set Resulti [£] , and 
does not contain Byzantine nodes, a contradiction is found in RemoveC ontradictions{) . Namely, the if 
conditions of line [24] holds. 

Note that, p may not be a correct path even though it contains no Byzantine nodes. For example p may 
contain nodes p z that are not even in the communication graph, i.e., p z eP\(CU B). 

Let p r G C U B be the first correct node on path p. Such a node exists, because pi is correct and on the 
path p. Since p r is correct, after the execution of ComputeResults{), we have that p r 's neighborhood, N r , 
is encoded in Resulti[r], see Lemma [6] 

Denote the last edge in the path (p T ,p s ), where p s G P \ (C U B). Note that node p s is not a node in 
the system and since Resulti [r] encodes JV r 's neighborhood, we have that p s Resulti [r] . Thus, the edge 
(p r ,Ps) is contradicting with the set Resulti[r]. Namely, by the condition in line |24l we have that line [25] 
must decrease Count[£]. 

We note that immediately before the function RemoveContradictionsQ returns, the integer Count[£] 
may count only incorrect paths, which contain at least one Byzantine node. Since there are at most / 
Byzantine nodes, Count[£] < f as needed. ■ 

Theorem [8] demonstrates the self-stabilization properties. 

Theorem 8 (Self-stabilization) Let R be a fair execution of Algorithm\l}that starts in an arbitrary config- 
uration and pi G C be a correct node. Within 0{\C U B\) asynchronous rounds, the system reaches a safe 
configuration after which pi 's output is always legal, see Definition \3\ 

Proof. The systems reaches configuration c G R of Lemma [6] within 0(\C U B\) asynchronous 
rounds. We show that c is a safe configuration by showing that the output is legal, we must show 
that ConfirmedTopologyi encodes a graph Goutput = {Vout,E out ), such that: (1) C C V out , (2) 
(En(CxC)) C Eout, (3) V out C CUB C iV,and(4) E out C (E n (C x C)) U (B x (CU B)) CPxN. 

For every correct node pk G C, we have that JVj. is confirmed in c, see Lemma [6] Thus, pk G V ou t and 
condition (1) holds. 
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Let (pj,Pk) be an edge in the communication graph between two correct nodes, we show (pj,Pk) E 
E out . Since pj is correct, it is inserted to ConfirmedTopologyi, see Lemma [6] Thus, (pj,Pk) € 
edges(Nj) A edges(Nj) C ConfirmedTopologyi in c, thus condition (2) holds as well. 

There is no p£ 6 P \ (C U -B) and node € AT, such that (pePj) € ConfirmedTopologyi, see 
LemmaEl Thus, F out C CUB C iV and E out Q(En{Cx C)) U(Bx(CUB)) CPxJV. I.e., conditions 
(3) and (4) hold in c. ■ 



B Correctness of Algorithm |2] 

Lemma|9] shows that senders and receivers can eventually find at least 2/ + 1 vertex-disjoint paths between 
them. Note that at least / + 1 of them are correct. 

Lemma 9 Let R be a fair execution ofAlgorithm\2\that starts in an arbitrary configuration and p s ,p r € C 
a pair of correct nodes (sender and receiver). Within 0(\C U B\) asynchronous rounds the system reaches 
a configuration in which the set ConfirmedTopology U Suspicious Edges encodes a set of If + 1 vertex 
disjoint paths from p s to p r and at least f + 1 of them are correct. 

Proof. Let c be a safe configuration with respect to Algorithm [T] Let Paths = 
getDisjointPaths(ConfirmedTopology U Suspicious Edges(),i, Destination) be a set of disjoint 
paths in c, as in line [25] where i € {s,r}. We first show that [ Paths [> 2/ + 1 before showing that 
at least / + 1 of them are correct. 

We consider the graph G' = (N, Eqi), which is computed from ConfirmedTopology and the suspi- 
cious edges in c. We demonstrate that C contains the real communication graph, G. Let e = (pj ,pk) € Eqi . 
When pj and pk are both correct, e € C since c is safe. When pj is correct and pp, is Byzantine, we must 
consider the cases in which p^ reports, and does not report, e as part of its local neighborhood. Namely, 
either e G ConfirmedTopology, or e € Suspicious EdgesQ, because pk does not report about e, but pi 
does. Since G C G' , G' must contain 2/ + 1 vertex disjoint paths between any p s and p r , because G does. 
Thus [ Paths |> 2/ + 1. 

Moreover, the same arguments implies that there may be at most / incorrect paths, which contain at 
least one Byzantine node. Hence, there are at least / + 1 correct nodes in Paths. ■ 

Definitions l8l l9l and [lOl are needed for lemmas ITT1 [121 and [131 

Definition 8 (Confirmation) Given configuration c, we say that message m is confirmed (by the receiver) 

when m E OutputMessageQueue. 

Definition 9 (Approve) Given fair execution, R, of Algorithm [2] we say that message m = (Source, 
Destination, VisitedPath, IntentedPath, ARQLabel, DATA, Payload) is being approved (by the 
sender p source) during the first atomic step, a sen d er > in which the sender executes line [iS] where Source = 
sender ARQLabel = msg sen der-ARQLabel and Payload = msg sen der-Po,yload, see line U7\ Denote 
by Capproved the configuration that immediately follows a sen dev Given configuration c that appears after 
Capproved in R, we say that message m is approved (by the sender) in configuration c. 

Definition 10 (Clear-sender-receiver) Given configuration c, we say that the sender is clear (with respect 
to the receiver), if the queue Confirmations[receiver] = in c. Moreover, the receiver is clear (with 
respect to the sender) , if the queue ReceivedM essages[sender] = $ inc. 
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Claim [10] shows that a message that is relayed on a correct path is received at the destination within 
0(\C U B\) asynchronous rounds. Moreover, the destination receives the message with correct visiting set. 

Claim 10 Let R be a fair execution of Algorithm^ that starts in a safe configuration, c, with respect to 
Algorithm\J} Let Psource, Pdest £ C be pair of correct nodes. Let c sen d be the configuration immediately 
following a step in which p SO urce sends message Msg on a correct path Path = p S ourceiPi,P2, ■ ■ -Pdest 
from source, p source, to destination, pdest- Within 0(\C U B\) asynchronous rounds, pdest receives Msg 
with a visiting set containing all nodes on Path except pdest- 

Proof. Upon the arrival of message m to pk (line [6]), node pi asserts that he is not the destination, pdest, 
(line IT]). Immediately after, p,, sends the message m to the next neighbor, pj+i, see line [9] Since the same 
argument holds when pj sends m to the next node in path, we have that within \Path\ asynchronous rounds, 
m is delivered to node pdest- B 

Claim [TTIsays that when the sender repeatedly sends message Msg, for a duration of at least 0(\CUB\) 
asynchronous rounds, the receiver eventually confirms message Msg. 

Claim 11 Let R be a fair execution of Algorithm^ that starts in a safe configuration, c, with respect to 
Algorithm^ Let p s ,p r G C be a pair of correct sending and receiving nodes. Suppose that, for a duration 
of at least O {capacity ■ \C U B\) asynchronous rounds, p s 's steps include only the execution of the func- 
tion ByzantineFaultToleranceSend(Msg) in the loop ofline\5\ Within that period, the system reaches 
configuration c rece i ve in which p r confirms Msg. 

Proof. Denote c sen d as the configuration immediately following the first step in which p s sends message 
Msg in R, see line [28] Within O (capacity ■ \C U B\) asynchronous rounds, the first frame containing 
Msg arrives at p r , see Claim [TO] Moreover, after another O (capacity ■ \C U B\) asynchronous rounds, 
every correct path relays message Msg at least O (capacity ■ \C U B\) times. This is correct since every 
asynchronous round, p s sends a new frame containing Msg on each of the 2/ + 1 vertex disjoint paths. 
Moreover, by Claim [TOl the last frame sent on all 2/ + 1 paths arrives after another 0(capacity • |C U B\). 

Assume, in the way of proof by contradiction, that Msg is not confirmed by p r . This implies that 
the queues, ReceivedMessages[p s ][*], in p r containing messages sent from p s were not cleared at least 
since c sen d, see line [22] Thus, p r contains capacity ■ n + 1 indications of Msg on / + 1 vertex disjoint 
paths. Denote ci ast as the configuration immediately after the arrival of the (capacity ■ n + l)-th frame of 
the / + l'th path to relay capacity ■ n + 1 frames containing Msg. Immediately after ci ast , p s must go 
through line [12] because the conditions in line[l2]hold. Thus, a contradiction and Msg is confirmed within 
0(capacity ■ \C U B\) asynchronous rounds. ■ 

Claim [12] says that when the receiver is sending acknowledgments about a message, that message even- 
tually becomes approved. We note that Claim [12] considers acknowledgments sent from the receiver to the 
sender, rather than messages sent from the sender to the receiver, as in Claim [TT] 

Claim 12 Let R be a fair execution of Algorithm^ that starts in a safe configuration, c, with respect to 
Algorithm [7] Let p s ,p r £ C be a pair of correct sending and receiving nodes. Suppose that, for a duration 
of at least O (capacity ■ \C U B\) asynchronous rounds, p r 's steps include only the execution of the function 
ByzantineFaultToleranceSend(Ack) in the loop ofline\23\ That is, p r is sending acknowledgments on 
message Msg. Within that period, the system reaches configuration c rece i ve in which p s approves Msg, see 
Definition [9] 
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Proof. Denote c sen d as the configuration immediately following the first step in which p r sends acknowledg- 
ment Ack in R, see line [23] Within O (capacity ■ \C U B\) asynchronous rounds, the first frame containing 
Ack arrives at p s , see Claim [TO] Moreover, after another 0(capacity • \C U B\) asynchronous rounds, 
every correct path relays message Ack at least Oicapacity • \C U B\) times. This is correct since every 
asynchronous round, p r sends a new frame containing Ack on each of the 2/ + 1 vertex disjoint paths. 
Moreover, by Claim [TO] the last frame sent on all 2/ + 1 paths arrives after another O (capacity • \C U B\). 

The queues, Confirmations[p r ] [*] are cleared only when a message sent to p r is approved, see line [2] 
Since, p r is acknowledging the current message, Msg, by sending Ack, the only message that can be ap- 
proved is Msg. This is true since each path, Path, may contain at most capacity - \ CL)B\ acknowledgments 
for other messages in the path queues. 

Assume, in the way of proof by contradiction, that Msg is not approved by p s . By the arguments above, 
p s 's queues, Confirmations s \p r ][*\, which contains p r 's acknowledgments that p s received, were not 
cleared at least since c sen d, see line[2l Thus, p s contains capacity ■ n + 1 indications of Ack on / + 1 vertex 
disjoint paths. Denote ci ast as the configuration immediately after the arrival of the (capacity ■ n + l)-th 
frame of the / + l'th path to relay capacity ■ n + 1 frames containing Ack. Immediately after ci as t, Ps 
must go through line [HO because the conditions in line[l8]hold. Thus, a contradiction and Msg is approved 
within 0(capacity • |C U B\) asynchronous rounds. ■ 

Lemma [13] shows that the senders repeatedly fetch messages. 

Lemma 13 Let R be a fair execution of Algorithm\2\that starts in a safe configuration, c, with respect to 
Algorithm\J\ Letp s ,p r € C be pair of correct sending and receiving nodes. Moreover, eg is the configuration 
that immediately follows the t-th time in R in which p s fetches a message from the input queue. For every I, 
the system reaches ci within 0(1 ■ \C U B\) asynchronous rounds. 

Proof. By the code of Algorithm [2] on every iteration of the do forever loop (lines [2] to [5]), a message is 
fetched in line [3] This do forever loop includes another loop in line [5] We prove the lemma by showing that 
the loop of line[5]is completed within 0(\C U B\) asynchronous rounds. 

The proof considers the case in which the sender, p s , does not wait in line [5] for a long time before 
considering the case in which p s does wait. We show that for the latter case, the receiver, p r , confirms p s 's 
current message. After confirming the message, the receiver, p r , begins sending acknowledgments to the 
sender, p s . The proof shows that after the acknowledgments are sent, p s approves the message and fetches 
a new one. We show this by considering the case in which p r repeatedly sends acknowledgments for a 
sufficient amount of time, and a case in which it does not. 

Suppose that p s does not wait in line[5]more than 0(capacity ■ \C U B\) asynchronous rounds. In this 
case, p s starts the infinite loop again within 0(capacity ■ \C U B\) asynchronous rounds, and fetch a new 
message, see line [3] Thus, for the case in which p s does not wait in line [5] more than O (capacity • |C U B\) 
asynchronous rounds, the lemma is correct. 

Suppose that p s is executing line [5] and waits for acknowledgments on message Msg for more than 
0(capacity ■ \C U B\) asynchronous rounds. Thus, p s floods 2f + 1 vertex-disjoint paths with the message 
Msg, see Claim[9l Eventually, the receiver, p r , receives message Msg for 0(capacity ■ \C U B\) times 
on / + 1 vertex-disjoint paths and confirms Msg, see Claim [TTJ After confirming it, the receiver sends 
acknowledgments on 2/ + 1 vertex-disjoint paths until confirming a new message Msg new . This is true 
because the condition in line [23] holds only when a new message is confirmed, see line [14] 

Let us consider the case in which, during O (capacity ■ \ CUB\) asynchronous rounds, message Msg new 
does not arrive to the receiver. By Claim [12] eventually the sender receives the acknowledgments for 
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capacity ■ n + 1 times on / + 1 vertex disjoint paths. Claim [12] also says that the sender considers the 
message accepted by the receiver. In line [HO the sender assigns Approved = true. Thus, the condition in 
line [5] holds and the sender fetches the next message, see line [3] Hence, the system reaches configuration 
c f e tch that immediately follows a step in which the sender, p s , fetches the next message. Thus, for the case in 
which, during Oicapacity ■ \CUB\) asynchronous rounds, message Msg new does not arrive to the receiver, 
the lemma is correct. 

We continue by considering the case in which, during 0(capacity • |C U B\) asynchronous rounds, 
message Msg new does arrive to the receiver. Let c con / be the configuration that immediately follows the 
step in which p r confirms Msg. Since the receiver confirms M sg, we have that p r is clear (with respect to 
the sender) in configuration c con f, see Definition [lO]and linel22l 

If Msg new was sent by the sender, it must have been fetched after c, and c f e t c h is reached when message 
Msg new is fetched. It may be the case however, that Msg new was not sent by the sender. Message Msg new 
was confirmed by 2f + 1 vertex disjoint paths. Since there are at most / Byzantines, at least one of these 
paths, Path, must be correct. Moreover, in c con f, the receiver is clear, thus the capacity ■ n + 1 that p r 
counts in ReceivedM essages[p s ][*] have all been received after configuration c con f. Note that the sender 
sends at least one of these messages, because at most capacity ■ n messages could be in the edges of Path at 
any given configuration. Thus the sender sends Msg new , which p s fetches immediately before cj e t c h- Le., 
the system reaches Cf etc h- ^ 

Theorem [8] says that, starting from that fourth (or even the third) message that the sender fetches, the 
receiver confirms the sender's messages. The proof of Theorem [8] is based on Lemma [141 which says that, 
in every sequence of four messages that the sender is fetching, the receiver confirms the fourth (or even the 
third) message. 

Lemma 14 Let R be a fair execution of Algorithm\2\that starts in a safe configuration, c s t a rt, with respect 
to Algorithm^ Let Ch be a configuration that immediately follows the h-th step in which the sender fetches 
the h-th input queue message, m^. Within 0(\C U B\) asynchronous rounds, the receiver confirms message 

Proof. 

Claim 15 In C2, the sender is clear ( with respect to the receiver), see Definition \TU\ 

Proof. By definition, C2 immediately follows atomic step 02, in which, after clearing the confirmation queue 
in line [2 the sender fetches message 1112 and sends it. □ 

Claim 16 Between the configurations C3 and 04, there is a configuration c rece iver-clear in which the receiver 
is clear ( with respect to the sender). 

Proof. Suppose, without the loss of generality, that immediately after c sen rf er _ c ; ear , the sender is waiting 
for a message with label 1. By lemma IT~3l the sender eventually fetches the next message. The sender 
can only fetch a new message once Approved is true, see line [5] Moreover, Approved is only set to true 
once the queue Confirmations[receiver][*] contains 2/ + 1 flooded paths, see line[l8] Thus, the sender 
counts 2f + 1 vertex disjoint paths that relayed acknowledgments with label 1. Moreover, the sender is clear 
in c sen der- dear- Hence, configuration c sen der-dear contains no message in Confirmations[receiver][*]. 
Starting from c sen d e r- clear . the sender receives capacity ■ n + 1 acknowledgments on 2/ + 1 vertex disjoint 
paths for the current message with label 1. Note that at least one of these 2/ + 1 paths, Path, is correct, 
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because there are / Byzantine. Since \Path\ < n and each edge on Path may contain at most capacity 
messages, we have that at least one of the acknowledgments that includes Path as its visiting path, is sent 
by the receiver between c sender - c i ear and configuration c receiver - send € R. We show that c receiver - send = 

Creceiver— clear- 

This means that after c Bender - c iear> the sender clears the confirmations queue, 
Confirmations[receiver][*], and fetches the next message, assigning it the label 2, see lines [2] 
through line [5] By similar arguments, we know that the receiver sends at least one acknowledgment with 
label 2. 

To conclude, there is a configuration c € R in which the receiver is sending acknowledgments with 
label 1, and then a configuration d in which the receiver sends acknowledgments with label 2. Moreover, 
between two consecutive executions of line [23] the receiver has to go through line [22] Thus, the receiver 
cleared it's message queues, Confirmations[sender][*], immediately before configuration c rece i ver _ c \ ear 

and C rece i ver — send — C rece i ver — c l ear . LJ 

Let us consider configuration c rece i ver - c i ear from the end of proof of Claim [T6l 

The next message to be sent after c rece i ver _ c i ear , is m^, the message fetched in C4, with label 0. Between 
Creceiver-ciear an d C4, all messages sent by the sender have the label 2. By arguments stated above, the 
message, m, that is the next message to be confirmed after c rece i ver - c i ear , must have been sent by the sender 
at least once since c rece i ver _ c i ear . The sender, sends only messages with label and 2. Moreover, the last 
message to be confirmed had a label 2. Thus, CurrentLabel = 2, see line EE] Any sent message with 
label 2 is not inserted to the confirmations queue, Confirmations[sender]{*} between c rece i ver _ c i ear and 
the configuration that immediately follows the next sender's fetch, see line [20] Thus, by line 0] the next 
message to be confirmed is a message with label 0, which must be 7714. ■ 

Theorem [8] (Self-stabilization) Let R be a fair execution of Algorithm\2\that starts in an arbitrary configu- 
ration. Within 0{\C U B\) asynchronous rounds, the system reaches a safe configuration c after which: (I) 
the receiver confirms message m in step a r r n £ R, and (2) for every step a™, there is a corresponding step, 
a™" £ R, that occurs before a™ and in which the sender sends m. 

Proof. Let c be the configuration that Claim [T6ldenote as C4, which the system reaches within 0(\C U B\) 
asynchronous rounds, see Lemma [13] Let m; be the i-th message fetched. 

Suppose that i > 4. Lemma [JJ] considers the four consecutive messages mj_3, . . . mi and says that the 
receiver confirms message mj. Thus, condition (1) holds. 

Condition (2) follows from arguments similar to the ones used in the proof of Lemma QT| Namely, 
for the case of % > 5, message m^-i is confirmed, see lemma [T4l Immediately after the receiver con- 
firms rrii-x, it clears the queue ReceivedMessages[sender][*], see lines l20l to l22l Thus, there exists 
a configuration c rece i ver - c i ear in which the receiver is clear (with respect to the sender) before a, see 
Definition [10] Moreover, a message is confirmed only if the queue ReceivedMessages[sender][*\ con- 
tains 2/ + 1 flooded paths, see line [12] These flooded paths implies that in configuration a, the queue 
ReceivedMessages [sender] [*] contains capacity ■ n + 1 indications of rrn on 2/ + 1 node disjoint paths. 
Thus, m-i is confirmed only after a period that follows c rece i ver - c i ear and includes its reception at least 
capacity ■ n + 1 times on each of the 2f + 1 vertex disjoint paths. 

Recall that we assume that there are at most / Byzantine nodes in the system. At least one path, Path, 
of the above 2/ + 1 paths is correct. Moreover, \Path\ < n and each edge on Path may contain at most 
capacity messages. Thus, at least one of the capacity ■ n + 1 message that were relayed on the correct path 
Path was sent by the sender. This completes the correctness proof. ■ 
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